Customization of gui layout based on history of use

ABSTRACT

A hand-held electronic device, method of operation and computer readable medium are disclosed. The device may include a case having one or more major surfaces. A visual display and a touch interface are disposed on at least one of the major surfaces. A processor is operably coupled to the visual display and touch screen. Instructions executable by the processor may be configured to a) present an image on the visual display containing one or more active elements; b) correlate one or more active portions of the touch interface to one or more corresponding active elements in the image on the visual display; and c) adjust a layout of content shown on the display according to a probability of one or more actions that may be taken with the one or more active elements.

CLAIM OF PRIORITY BENEFIT

This application claims the priority benefit of U.S. Provisional PatentApplication No. 61/180,400, filed May 21, 2009, the entire contents ofwhich are incorporated herein by reference.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09019US00), to CharlesNicholson and Gary M. Zalewski entitled “CONTINUOUS AND DYNAMIC SCENEDECOMPOSITION FOR USER INTERFACE” filed the same day as the presentapplication, the entire contents of which are incorporated herein byreference.

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09020US00), to CharlesNicholson and Gary M. Zalewski entitled “HAND-HELD DEVICE WITH ANCILLARYTOUCH ACTIVATED ZOOM” filed the same day as the present application, theentire contents of which are incorporated herein by reference.

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09021US00), to CharlesNicholson and Gary M. Zalewski entitled “HAND-HELD DEVICE WITH ANCILLARYTOUCH ACTIVATED TRANSFORMATION OF ACTIVE ELEMENT” filed the same day asthe present application, the entire contents of which are incorporatedherein by reference.

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09022US00), to CharlesNicholson and Gary M. Zalewski entitled “TOUCH SCREEN DISAMBIGUATIONBASED ON PRIOR ANCILLARY TOUCH INPUT” filed the same day as the presentapplication, the entire contents of which are incorporated herein byreference.

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09023US00), to CharlesNicholson and Gary M. Zalewski entitled “TOUCH CONTROL WITH DYNAMICALLYDETERMINED BUFFER REGION AND ACTIVE PERIMETER” filed the same day as thepresent application, the entire contents of which are incorporatedherein by reference.

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09024US00), to CharlesNicholson and Gary M. Zalewski entitled “TOUCH SCREEN DISAMBIGUATIONBASED ON PRIOR ANCILLARY TOUCH INPUT HAND-HELD DEVICE WITH TWO-FINGERTOUCH TRIGGERED SELECTION AND TRANSFORMATION OF ACTIVE ELEMENTS” filedthe same day as the present application, the entire contents of whichare incorporated herein by reference.

This application is related to commonly assigned co-pending applicationSer. No. 12/______, (attorney docket number SCEA09043US00), to CharlesNicholson and Gary M. Zalewski entitled “DYNAMIC RECONFIGURATION OF GUIDISPLAY DECOMPOSITION BASED ON PREDICTIVE MODEL” filed the same day asthe present application, the entire contents of which are incorporatedherein by reference.

FIELD OF THE INVENTION

Embodiments of the present invention are related to handheld devices andmore particularly to hand-held devices that utilize a visual display andtouch interface.

BACKGROUND OF THE INVENTION

Handheld consumer electronic devices such as cellular telephones,portable internet devices, portable music players, and hand held gamingdevices often include some form of visual display, such as a flat screenvideo display or a touchscreen display. Touchscreens are displays whichalso have the ability to detect the location of touches within thedisplay area. This allows the display to be used as an input device,removing the keyboard and/or the mouse as the primary input device forinteracting with the display's content. Such displays can be attached tocomputers or, as terminals, to networks. Touchscreens also have assistedin recent changes in the design of personal digital assistant (PDA),satellite navigation and mobile phone devices, making these devices moreusable.

Touchscreens have become commonplace since the invention of theelectronic touch interface in 1971 by Dr. Samuel C. Hurst. They havebecome familiar in retail settings, on point of sale systems, onautomatic teller machines (ATMs) and on PDAs where a stylus is sometimesused to manipulate a graphical user interface (GUI) and to enter data.The popularity of smart phones, PDAs, portable game consoles and manytypes of information appliances is driving the demand for, and theacceptance of, touchscreens.

The visual displays used in hand-held devices are relatively smallcompared to computer screens or television screens. This often makes itdifficult to see information displayed on the screen. Some hand-helddevices allow the display to zoom-in on a selected portion of a largerimage so that the selected portion may be magnified and viewed ingreater detail. To implement such a zoom feature typically requires thehand-held device to implement some way of selecting the portion to bemagnified. Prior art solutions include the use of a touchscreen as thevisual display and software that allows the user to select the portionof the display to be magnified with his fingers or a stylus.Unfortunately, because the screen is small, the user's fingers oftenobscure the part that is to be selected making selection difficult.

It is within this context that embodiments of the present inventionarise.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1A is a schematic diagram illustrating a portion of contentcontaining active elements on a visual display of a handheld device.

FIG. 1B is a schematic diagram illustrating decomposition of the portionof content displayed on the device in FIG. 1A into sensitive regionscorresponding to active elements in accordance with an embodiment of thepresent invention.

FIGS. 1C-1I are schematic diagrams of possible handheld devices that maybe used in conjunction with embodiments of the present invention.

FIG. 2A is a side view diagram of a hand-held device according to anembodiment of the present invention.

FIG. 2B is a block diagram of a hand-held device according to anembodiment of the present invention.

FIG. 3A is a flow diagram of illustrating operation of a hand-helddevice according to an embodiment of the present invention.

FIG. 3B is a three-dimensional schematic diagram of a hand-held deviceillustrating magnification of a selected displayed active elementaccording to an embodiment of the present invention.

FIG. 4A is a flow diagram of illustrating operation of a hand-helddevice according to an embodiment of the present invention.

FIG. 4B is a three-dimensional schematic diagram illustrating selectionof an active element with a touch pad on a handheld device according toan embodiment of the present invention.

FIG. 4C is a plan view schematic diagram illustrating magnification ofan active element in response to activation of a corresponding region ofthe touch pad.

FIG. 4D is a three-dimensional schematic diagram illustrating selectivemagnification of a portion of content presented on a display of ahandheld device using a touch pad according to an embodiment of thepresent invention.

FIG. 5 is a plan view schematic diagram of a handheld deviceillustrating an example of transformation of an active element presentedon a visual display in accordance with an embodiment of the presentinvention.

FIGS. 6A-6E are plan view schematic diagrams of a hand-held device atdifferent stages of operation according to an embodiment of the presentinvention.

FIGS. 7A-7E are plan view schematic diagrams of a hand-held device atdifferent stages of operation according to an embodiment of the presentinvention.

FIGS. 8A-8C are plan view schematic diagrams of a hand-held devicehaving a touch screen at different stages of operation according to anembodiment of the present invention.

FIG. 9A is a flow diagram of illustrating operation of a hand-helddevice according to an embodiment of the present invention.

FIG. 9B is a plan view schematic diagram illustrating an example of howdecomposition of displayed content may change as probability ofsubsequent actions change.

FIG. 10A is a flow diagram of illustrating operation of a hand-helddevice according to an embodiment of the present invention.

FIGS. 10B-10C are plan view schematic diagrams of a hand-held device atdifferent stages of operation according to an embodiment of the presentinvention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

Continuous and Dynamic Scene Decomposition for User Interface

According to certain embodiments of the present invention, content to berendered on a hand held device may be decomposed into a number ofregions that fill the area of a display screen. Each region may beassociated with a different active element of the content that isdisplayed on the screen. These regions may be mapped to correspondingtouch-sensitive regions of a user interface on the device. Eachtouch-sensitive region corresponds to a different active element. Insome embodiments, the user interface may be a touch pad that is separatefrom the display screen. In other embodiments, the display screen may bea touch screen and the user interface may therefore be part of thedisplay screen. Depending on the type of touch screen, a user mayinteract with the touch screen with a touch of the user's finger or bytouching the screen with a stylus.

By way of example, and not by way of limitation, content, such as a webpage, rendered on a hand held device is decomposed into a number ofregions. Each region may be associated with a different active elementthat is displayed. An entire area of a touch pad on the back of thedevice may be divided into touch sensitive regions. Each touch sensitiveregion may correspond to a different active element. The displayed webpage can be broken into the active regions for the back touch byperforming a decomposition on the browser-rendered html canvas. A usercan then “select” one of the active elements shown on the front screenby touching the corresponding region on the back touch. Since each areaon the back touch is much larger than the displayed active element, theactive elements are easier to select using the back touch than with thefront touch screen.

By way of a more detailed example, content in the form of an htmldocument, such as a web page, may be decomposed into the active regionsfor the back touch by performing a mathematical decomposition referredto generally as a tessellation on the browser-rendered html canvas. Thehtml canvas determines how the html document is displayed on a screen.The tessellation divides the portion of the document that is to bedisplayed into a finite number of regions that divide up the area of thescreen. Each region corresponds to an active element in the portion ofthe document that is to be displayed on the screen. According to oneembodiment, these regions may be mapped to corresponding touch-sensitiveregions of a touch pad. A user can then “select” an active element shownon the front screen by touching the corresponding touch-sensitive regionon the touch pad. As a result of the tessellation, each touch-sensitiveregion may be significantly larger than the corresponding active elementdisplayed on the screen. Consequently, where the screen is a touchscreen, the active elements may be easier to select using the touch padthan with the touch screen.

There are a number of different ways in which the tessellation may beperformed. In general, it is preferable for the tessellation to divideup the displayed portion of the document into convex regions. By way ofexample, and not by way of limitation, tessellation of the displayedportion into convex regions may be implemented as a Voronoidecomposition, sometimes also called a Voronoi tessellation, a Voronoidiagram, or a Dirichlet tessellation. The Voronoi decomposition is akind of decomposition of a metric space determined by distances to aspecified discrete set of objects in the space, e.g., by a discrete setof points. In a simplest case, a plane contains a set of points Sreferred to as Voronoi sites. Each Voronoi site s has a Voronoi cell,also called a Dirichlet cell, V(s) consisting of all points closer to sthan to any other site. The segments of the Voronoi diagram (i.e., theedges of the Voronoi cells) are all the points in the plane that areequidistant to two sites. The Voronoi nodes (i.e., the corners of a cellwhere two edges meet) are the points equidistant to three (or more)sites.

Transformation of active elements may be context sensitive. For example,it may make sense to simply magnify a button to make it easier to usewith the touch screen. However, it may be more convenient to transform ascroll bar to a form that is more convenient for a user to manipulate ona touch screen.

As an example, a web page might normally appear on the device's screenas depicted in FIG. 1A. In this example a portion of a web page 101 isdisplayed on a visual display of a device 102. The web page 101 includesactive elements. As used herein, an active element refers to a portionof the displayed web page that a user may interact with through asuitable user interface. Non-limiting examples of active elementsinclude a radio button 104, a text entry box 106, a link 108 (e.g., anhtml link or web link), and a scroll bar 110. The web page may alsoinclude inactive content, such as normal text 112 or images.

As depicted in FIG. 1B, the area of the displayed web page 101 may bedecomposed into a radio button region 114 that corresponds to the radiobutton 104, a text entry box region 116 corresponding to the text entrybox 106, a link region 118 corresponding to the link 108, and a scrollbar region 120 corresponding to the scroll bar 110. It is noted thatthere is no region corresponding to the text 112 since the text is notan active element in this example. According to some embodiments of thepresent invention, the radio button region 114, text entry box region116, and link region 118 may be mapped to corresponding regions on atouch sensitive interface.

In some embodiments, the touch sensitive interface may be a touch screenthat is part of the visual display. Alternatively, the touch sensitiveinterface may be a touch pad that is separate and apart from the visualdisplay. There are a number of possible configurations for the visualdisplay and touch pad. Possible examples of such configurations includeone in which a visual display V and a touch pad T are on the same sideof a case C, as shown in FIG. 1C, on adjacent sides of the case C, asshown in FIG. 1D, or on opposite sides of the case C as shown in FIG.1E. Other examples include configurations in which the visual display Vand touch pad T are located on separate case portions C₁ and C₂respectively. By way of example, and not by way of limitation, the caseportions C₁, C₂ may be connected to each other in a slidingconfiguration, as shown in FIG. 1F, in a hinged configuration as shown,e.g., in FIG. 1G, FIG. 1H, or FIG. 1I. In FIG. 1F, the visual display Vand touch pad T face inward when case portions C₁ and C₂ are in a closedposition. Alternatively, as shown in FIG. 1G, the visual display V mayface outward and the touch pad T may face inward (or vice versa) whenthe case portions C₁ and C₂ are in a closed position. Furthermore asshown in FIG. 1H, the visual display V and touch pad T face outward whencase portions C₁ and C₂ are in a closed position.

According to an embodiment of the invention, a shown in FIG. 2A, ahandheld electronic device 200 may include a case 201 with a visualdisplay 202 located on a major surface 225A of the case 201 referred toherein as the front surface. A touch pad 204 may be located on anothermajor surface 225B of the case 201 (referred to herein as the backsurface) that is opposite the front surface. The case may be ofsufficiently small size that it can be held in a user's hand.

As seen in FIG. 2B, the device may include a controller 203, thecomponents of which may be located within the case 201. The controller203 includes a processor 207 operably coupled to the visual display 202and the touch pad 204. In some embodiments, the device 200 may includemultiple processors 207 if parallel processing is to be implemented. Thedevice 200 may be configured for use as a game device, a phone, aportable media player, an email device, web browser device and the like.

The hand-held device 200 may also include well-known support functions,such as input/output (I/O) elements 211, power supplies (P/S) 213, aclock (CLK) 215 and cache 217.

The device 200 may optionally include a mass storage device 219 such asa disk drive, CD-ROM drive, flash drive, or the like to store programsand/or data. The touch screen 202, touch pad 204, processor 207, memory208 and other components of the device 200 may exchange signals (e.g.,code instructions and data) with each other via a system bus 220 asshown in FIG. 2B. In some embodiments, the device 200 may include anetwork interface 216, configured to allow the device to exchangesignals with other devices over a network. Furthermore, the hand-helddevice 200 may include one or more sensors 218. Such sensors mayinclude, e.g., an inertial sensor such as an accelerometer or tiltsensor, an optical sensor, an acoustic sensor such as a microphone ormicrophone array. The sensors may generate inputs to the programinstructions 210 that reflect the environment in which the hand-helddevice operates.

The visual display 202 may be any suitable form of display capable ofpresenting visible symbols and/or graphical images. By way of examplethe visual display 202 may include a flat panel display, such as aliquid crystal display (LCD) or light emitting diode (LED) display. Insome embodiments, the visual display 202 on the front surface may alsoincorporate a touch pad to provide an interface for receiving usercommands. In some embodiments the touch pad 204 may optionally include avisual display. The touch pad 204 on the back surface may be based onany suitable touch screen technology, such as resistive,surface-acoustic wave (SAW) capacitive, infrared, strain gauge, opticalimaging, dispersive signal technology, acoustic pulse recognition,frustrated total internal reflection or graphics tablet based onmagneto-strictive technology that responds to the proximity of a user'sfingers. Any of these same technologies may also be incorporated intothe visual display 202 on the front surface if desired. In a preferredembodiment, the visual display 202 includes a resistive touch screencoupled to the controller 203 and the touch pad 204 includes acapacitive touch screen.

By way of example a resistive touch screen panel may be composed ofseveral layers including two thin metallic electrically conductive andresistive layers separated by thin space. When some object touches thiskind of touch panel, the layers are connected at a certain point. Thepanel then electrically acts similar to two voltage dividers withconnected outputs. This causes a change in the electrical current whichis registered as a touch event which may be sent to the processor 207for processing.

Surface Acoustic Wave technology uses ultrasonic waves that pass overthe touch screen panel. When the panel is touched, a portion of the waveis absorbed. This change in the ultrasonic waves registers the positionof the touch event and sends this information to the controller forprocessing.

A capacitive touch screen panel may be coated with a material, e.g.,indium tin oxide that conducts a continuous electrical current acrossthe sensor. The sensor therefore exhibits a precisely controlled fieldof stored electrons in both the horizontal and vertical axes. When thecapacitive touch screen's ‘normal’ capacitance field (its referencestate) is altered by an externally applied electric field, e.g., from auser's finger, electronic circuits located at each corner of the panelmeasure a resultant ‘distortion’ in the characteristics of the referencefield and send the information about the event to the processor 207 formathematical processing.

An infrared touch screen panel may employ one of two differentmethodologies. One method uses thermal induced changes of the surfaceresistance. Another method is an array of vertical and horizontal IRsensors that detected interruption of a modulated light beam near thesurface of the screen.

In a strain gauge configuration the screen is spring mounted on the fourcorners and strain gauges are used to determine deflection when thescreen is touched. This technology may also measure movement of thescreen 202 along the Z-axis.

In touch screen technology based on optical imaging, two or more imagesensors may be placed around the edges (mostly the corners) of thescreen. Infrared backlights may be placed in a camera's field of view onthe other sides of the screen. A touch shows up as a shadow and eachpair of cameras can then be triangulated to locate the touch.

Dispersive signal technology may use sensors to detect mechanical energyin the glass that occurs due to a touch. Complex algorithms theninterpret this information and provide the actual location of the touch.

Touch screens based on acoustic pulse recognition may use more than twopiezoelectric transducers located at some positions of the screen toturn the mechanical energy of a touch (vibration) into an electronicsignal. This signal may then be converted into an audio file, and thencompared to preexisting audio profile for every position on the screen.

Touch screens based on frustrated total internal reflection use theprinciple of total internal reflection to fill a refractive medium withlight. When a finger or other soft object is pressed against thesurface, the internal reflection light path is interrupted, making thelight reflect outside of the medium and thus visible to a camera behindthe medium.

In some embodiments, the device 200 may include one or more optionalbuttons coupled to the controller 203 to provide additional sources ofinput. There are a number of different possible locations for theoptional buttons 206. By way of example, and without loss of generality,one or more optional buttons 206 may be located on the front surface225A, the back surface 225B, along a side edge 222 of the device 200 oron a beveled edge 224.

The hand-held device 200 may further include a memory 208 (e.g., RAM,DRAM, ROM, and the like). A computer readable medium such as the memory208 may store program instructions 210 for execution on the processor207. The program instructions 210 may be configured to respond to inputsfrom one or more input sources on the device (e.g., the visual display202, the touch pad 204, or buttons 206) or from remote input sourcesthat are coupled to the device. The program instructions 210 may includedisplay driver instructions 212 configured to generate images displayedon the visual display 202. The program 210 may include touch pad driverinstructions 213 that respond to inputs received from the touch pad 204.It is noted that in some embodiments, the functions of the visualdisplay 202 and touch pad 204 may be combined into a single touch screeninterface that may serve as both an input and an output device.

Hand-Held Device with Ancillary Touch Activated Transformation of ActiveElement

By way of example, and not by way of limitation, in one version of thisembodiment, a hand-held electronic device may have a case with one ormore major surfaces. A visual display may be disposed on at least one ofthe major surfaces. A touch interface may be disposed on at least one ofthe major surfaces. A processor may be operably coupled to the displayand the touch interface. An image containing content may be rendered onthe display. The content may be divided into a number of regions. Eachregion may be associated with a different active element such as a linkor check box that is displayed. The entire area of a touch interface maybe divided into touch sensitive regions. Each touch sensitive region maycorrespond to a different active element shown on the display. A usermay select an active element by touching the corresponding region on thetouch interface. When the active element is selected its appearanceand/or operation may be transformed so that the element is easier tomanipulate with the touch interface. The transformation may be animatedso that the user can easily see which active element is beingtransformed. After the user interacts with the transformed activeelement, the element may revert to its original form by a reverseanimation.

There are a number of different ways in which the reversion of atransformed element may be triggered. By way of example, and not by wayof limitation, if the transformation is triggered by a user's touch on aregion of the touch interface 204 corresponding to an active element,the reversion may be triggered by removal of the touch.

By way of example and not by way of limitation, the program 210 mayfurther include transformation instructions 214, which may beconfigured, e.g., by appropriate software programming, to operate thedevice 200 according to a method illustrated generally in FIG. 3A. Asindicated at 302 in FIG. 3A, a portion of content to be displayed on thedisplay 202 may be decomposed, e.g., by Voronoi composition, asdiscussed above. Active elements within the displayed portion may becorrelated corresponding portions of a touch interface, as indicated at304. The touch interface may be the touch pad 204 or the visual display202, if it includes a touch screen. As a user manipulates the touchinterface, the program 210 may determine whether the user has selectedany portion of the touch interface that corresponds to an activeelement, as indicated at 306. If the user selects one of these activeportions, a transformed view of the corresponding active element maythen be presented on the visual display 202 as indicated at 308.

There are a number of ways in which an active element may betransformed. For example, as illustrated in FIG. 3B, an icon 312representing an active element may simply be presented in magnified form314. This allows the magnified form 314 to be more easily manipulated bythe user if the visual display 202 is a touch screen. The transformationof the selected active element may be animated so that the user caneasily see which active element is being transformed. After the userinteracts with the transformed active element, the element may revert toits original form by a reverse animation. The re-transformation may alsobe animated. There are a number of variations on the transformation ofactive elements within the scope of embodiments of the presentinvention. A number of these are discussed below.

Hand-Held Device with Ancillary Touch Activated Zoom

In this embodiment, a hand-held electronic device may have a case withone or more major surfaces. A visual display may be disposed on at leastone major surface. A touch pad may be disposed on at least one of themajor surfaces. A processor may be operably coupled to the visualdisplay and the touch screen. Instructions executable by the processormay be configured to: a) present an image on the visual display; b)identify an active portion of the touch pad in response to userinteraction with the touch pad; c) correlate the active portion of touchpad to a corresponding region of display; and d) present a magnifiedview of the corresponding region on the visual display. As an example, auser may slide his finger over touch pad on a back side of the device.The location of the user's finger may be mapped to a correspondingregion of the display on the front side. A magnified view of this regionmay be displayed in a window on the display.

In some versions of this embodiment, the transformation instructions 214may be configured to track a user's manipulation of the touch pad 204 onthe back surface as indicated at 402 in FIG. 4A. A magnified view of acorresponding portion of an image may be presented on the visual display202. For example, as shown in the flow diagram of FIG. 4A the program210 may track a user's manipulation of the touch pad 204, as indicatedat 402, e.g., using the touch pad driver 213. Specifically, the touchpad driver 213 may determine which portion 412 of the touch pad 204 hasbeen activated by a user's touch, as shown in FIG. 4B. The activeportion 412 may be correlated to a corresponding region of the visualdisplay 202, as indicated at 404. A magnified view of the content withinthe corresponding region 414 may be displayed on the display 202 asindicated at 406.

In some embodiments, the touch pad 204 may be tessellated into regionsthat correspond to active elements shown on the display 202. When a useractivates one of the regions of the touch pad 204 that corresponds to anactive element, that active element may be magnified on the touch screenas depicted in FIG. 4B. For instance, referring to the example describedabove with respect to FIG. 1A-1B, if the user presses the back touchregion 118 corresponding to the link 108, a magnified link 418 may bedisplayed on the touch screen as shown in FIG. 4C.

In alternative versions of embodiment described with respect to FIG.4A-4B, it is not strictly necessary to perform a tessellation or similardecomposition of displayed portion of content. Instead, the program 210may simply track the user's activation of a portion of the touch pad204, correlate the activated portion to a corresponding region ofcontent displayed on the screen and present a magnified view 414 of thecontent in the corresponding region as shown in FIG. 4D. This makes itmuch easier to see and user the selected active elements shown on thescreen. This also allows for an enhanced usability of the visual display202 in the case where the visual display is also a touch screen.

It is noted that within the context of embodiments of the presentinvention there are many alternative ways in which an active element maybe transformed once it has been selected through activation of acorresponding region of a touch interface. For example, in addition to,or instead of, altering the displayed size of an active element, theappearance and or nature of operation of an active element may bealtered by the transformation.

By way of example, consider the case shown in FIGS. 1A-1B. It may beimpractical to magnify the scroll bar 110. It may also be awkward tomanipulate the scroll bar displayed on a small touch screen in aconventional fashion by moving the small box or “clicking” on the up ordown arrows at the ends of the scroll bar. Instead of magnifying thescroll bar 110, a transformation may take place as depicted in FIG. 5.In this example, the scroll bar 110 may be transformed into atransformed scroll bar 130 that may operate differently from aconventional scroll bar. For example, the scroll speed and direction ofthe transformed scroll bar 130 may depend on where the user places acursor 132 within the scroll bar, e.g., using a stylus. This makes itmuch easier to use the scroll bar on a touch screen. There are a numberof ways in which the appearance and/or nature of operation of an activeelement may be transformed. For example, a check box may be transformedinto a toggle switch, which may be more intuitive to operate on ahand-held device.

Touch Screen Disambiguation Based on Prior Ancillary Touch Input

In this embodiment a hand-held electronic device may have a case withfirst and second major surfaces as discussed above. A touch screendisplay may be disposed on the first major surface and a touch pad maybe disposed on another major surface. An image containing content isrendered on the display. The content can be divided into a number ofregions. Each region may be associated with a different active element,e.g., as discussed above. An entire area of a touch pad may be dividedinto touch sensitive regions. Each touch sensitive region corresponds toa different active element shown on the touch screen. A user may selectan active element by touching the corresponding region on the touch pad.

As discussed above, when an active element is selected its appearanceand/or operation may be transformed so that the element is easier tomanipulate with the touch screen. As noted above, the transformation canbe animated so that the user can easily see which active element isbeing transformed. After the user interacts with the transformed activeelement, the element may revert to its original form by a reverseanimation. For example, when a user selects an active element bypressing the corresponding region on the touch pad, the active elementmay be magnified on a front display that also acts as a touch screen.After the user has manipulated the transformed active element, thetransformed element may revert to its normal appearance. However, theactive element that was manipulated may be highlighted on the touchscreen so that the user can tell which active element was most recentlychanged. If the user wants to re-use the highlighted active element,this element can be selected by pressing on the general area of theelement on the front touch screen. If the user's finger touches severalactive elements, this action may be disambiguated as an activation ofthe highlighted active element.

By way of example, and not by way of limitation, as normally displayed,content, such as a web page containing multiple active elements, e.g.,check boxes 602 may appear on a touch screen display 202 as shown inFIG. 6A. The area of the displayed page may be broken up into ninedifferent regions 612 as shown in FIG. 6B.

In this example, each check box region 612 is mapped to a differentcorresponding region of a touch pad (not shown). The touch pad may belocated on a different part of the device, e.g., on a back surface of acase 201 or in some other configuration, e.g., as depicted in FIGS.1C-1I. A user may “select” one of the nine check boxes shown on thetouch screen by touching the corresponding region on the touch pad.Since each area on the back touch is much larger than the displayedcheck boxes, the check boxes are easier for the user to select. When auser selects an active element by pressing the corresponding region onthe touch pad, the active element may be magnified or otherwisetransformed on the touch screen. The user can then more easily see theactive element and/or interact more easily with it using the touchscreen 202.

For example, if the user presses a touch pad region 612C correspondingto a center check box 602C, a magnified center check 622C box may bedisplayed on the touch screen 202 as shown in FIG. 6C. Portions of thedisplayed content that are associated with the text box, e.g.,associated text 624C may also be magnified so that they are easier toread.

If the user wants to check the center check box 602C, this box can beselected by pressing on the corresponding magnified check box 622C onthe touch screen 202, e.g., using a stylus or the user's finger. Afterthe user has selected the magnified center check box 622C, the magnifiedcheck box 622C may revert to its normal appearance on within thedisplayed content. However, the center check box 602 may be highlightedon the display 202 so that the user may easily perceive that the centercheck box was the one that was most recently changed as shown in FIG.6D.

The transform instructions 214 may filter input from the touch screendriver 213 based on the fact that the center check box 602C was theactive element that was most recently modified. For example, thetessellated regions corresponding to displayed active elements may bemapped to the touch screen 202 as well as the touch pad 204. Thetransform instructions 214 may be configured to filter signalscorresponding to touch activation of these regions such that a user'stouch of any one or more of these regions is interpreted as a touch ofthe most recently modified active element displayed. This way, anambiguous input on the touch screen may be disambiguated based onprevious user interaction with displayed active elements. Thus, e.g., ifthe user wants to un-check the center check box 602C, this box can beselected or magnified by pressing on one or more of the check boxregions 612 that are mapped to the touch screen 202. Even if the user'sfinger F touches several check box regions, as depicted in FIG. 6E, thetransform instructions 214 may interpret this action as a selection ofthe center check box 602.

There are a number of possible variations on the embodiment describedwith respect to FIG. 6A-6E. These variations may address unforeseenproblems associated with a handheld device that uses a touch screen onthe front of the device and a touch pad on the back of the device. Manyof these problems may arise since the user tends to primarily rely onthe front side touch screen since the view of the position of the user'sfingers on the backside touch pad is often obscured by the case.

Touch Control with Dynamically Determined Buffer Region and ActivePerimeter

This embodiment deals with the problem of how to select multiple activeelements on the touch screen of a hand-held device without the userhaving to hold down a control, e.g., on the backside touchpad, to keepthe selected elements magnified or enhanced.

Certain implementations of this embodiment may be based on a modaldialog box format that uses a non-responsive or inactive region thatwraps around a perimeter of a state input touch field, and, adynamically sized accept and cancel region that wraps around theinactive buffer region of the modal dialog box. It is noted that thisembodiment may be implemented in a device having a single touch screen.Although two touch interfaces are not necessary, this embodiment may beemployed in devices that have two touch interfaces, e.g., a front touchscreen and a back touch pad. In one example of this embodiment, selectand cancel regions may be dynamically set to a thickness based on thesize of a dialog box while preserving buffer region.

According to this embodiment, content rendered on a screen of a handheld device may be divided into a number of regions. Each region isassociated with a different active element such as a link or check boxthat is displayed. The entire area of a touch interface, e.g., a touchscreen onto which the content is rendered, or a separate touch pad onthe back of the device may be divided into touch sensitive regions. Eachtouch sensitive region corresponds to a different active element. A userselects an active element by touching the corresponding touch sensitiveregion. A transformed or magnified active element may be displayed onthe screen. The enhanced active element may be manipulated with thetouch interface, e.g., a front touch screen or back touch pad. A bufferregion surrounds the enhanced active element.

Nothing happens if the user touches this area of the touch interface.The remaining region of the touch interface outside the buffer region isrepurposed so that touching on this region can either commit to themanipulation of the active element or cancel the manipulation of theactive element.

By way of example, and not by way of limitation, content 701, such as aweb page, might normally be displayed a hand held device's touch screenas shown in FIG. 7A. In this example, the displayed content 701 includesa number of active elements, such as a radio button 702, a text entrybox 704, a link 706 and a group of check boxes 708 and inactiveelements, such as text 710.

As depicted in FIG. 7B, the active elements may be to correspondingtessellated regions of a touch screen or touch pad, e.g., as describedabove. Specifically, the displayed content 701 may be decomposed into aradio button region 712 that corresponds to the radio button 702, a textentry box region 714 corresponding to the text entry box 704, a linkregion 716 corresponding to the link 706, and a text box group region718 corresponding to the text box group 708. It is noted that there isno region corresponding to the text 710 since the text is not an activeelement in this example.

A user may select the check box group, e.g., by touching thecorresponding region 718 on device's the touch screen or a separatetouch pad. An enhanced (e.g., transformed or magnified) text box group728 may then be displayed on the touch screen, e.g., as shown in FIG.7C. Animation may be used to show the transformation or magnification ofthe text box group so that it will be clear that this is the particularactive element that was selected. The transformation or magnification ofthe text box group 708 allows a user to more easily activate selectedcheck boxes on the touch screen.

A buffer region 721 of the touch screen surrounds the enhanced check boxregion 728. The program 210 may be configured such that nothing happensif the user touches the buffer region 721. The remaining region 725 ofthe touch screen outside the buffer region 721 may be repurposed so thattouching on this region can either commit to the selected boxes orcancel the text box selection and make the text boxes go away. Forexample, as depicted in FIG. 7D, one side of the remaining region 725Aof the touch screen may be repurposed as a “commit” button and anotherside of the remaining region 725B may be repurposed as a “cancel”button. These regions may have different shading and icons may bedisplayed within each region as a guide to the user. Touching the“commit” region 725A commits the selected check boxes and reverts theenhanced text box region 728 to the original form of the text box group708. Touching the “cancel” region 725B cancels the selection of theselected check boxes and reverts the enhanced text box region 728 to theoriginal form of the text box group 708.

The thickness of the remaining region 725 may be dynamically determinedbased on a size of the enhanced check box region 728 in such a way thatthe buffer region 721 is preserved. By way of example, and not by way oflimitation, the buffer region 721 may be preserved it the thickness ofthe remaining region is adjusted so that the thickness of the bufferregion 721 around the enhanced check box region 728 is greater than orequal to some minimum thickness. The minimum thickness may be determinedempirically based on some criteria related to ease of use of the device.For example, a minimum thickness of the buffer region may be determinedto reduce the likelihood that a user will inadvertently touch the“commit” region 725A or cancel region 725B.

There are a number of possible variations on the embodiment describedabove with respect to FIGS. 7A-7D. In one variation, the hand helddevice may include an inertial sensor (e.g., an accelerometer or tiltsensor), which may be used to detect a tilt of the device to togglebetween commit and cancel. The user could then touch anywhere in there-purposed region 725 to activate the commit or cancel. Alternatively,the user may commit or cancel by pressing on a separate touch pad, ifthe device includes one. A “commit” or “cancel” icon may be displayed onthe touch screen depending on which way the device is tilted (asdetermined by a signal from the inertial sensor). According to anothervariation on this embodiment, the active element may be a color pickerhaving two modes of operation that may be selected using a re-purposedregion 725. For example, the color picker may have a default mode thatallows a user to select from among a limited number of colors (e.g., 16colors). By touching the re-purposed region 725 a user may elect toenter an expanded mode that allows the user to select from a greaternumber of colors (e.g., 256 colors). The repurposed region 725 mayinclude an icon or other indication to the user that this expanded modeis available.

Another variation addresses the situation where a user has someinformation copied to a temporary memory location sometimes called a“clipboard”. In conventional touch screen interfaces, if the user needsto enter information into the text entry box 704, the device convertspart of the touch screen to a virtual keyboard. In conventionalhand-held devices, the user must then enter the text with the virtualkeyboard. This is often inconvenient, particularly if text can be copiedusing the device.

As shown in FIG. 7E, according to an embodiment of the invention, bycontrast, a special icon 734 may be shown on the touch screen inconjunction with an enhanced text entry box 724 if the user selects thetext entry box region 714 and data has been copied that may be pastedinto the text entry box. In some embodiments portions of the touchscreen may be repurposed as a virtual keyboard 744, which may bedisplayed as part of or in conjunction with the enhanced text entry box724. The user may touch the icon 734 to paste the stored text into thetext entry box 704. The user may also enter text via the virtualkeyboard. The user may then return the device to normal operation, e.g.,through interaction with the touch screen or touch pad. For example, ifthe text entry box 704 is selected by touching the text entry box region714 using the touch screen, the user may touch the text entry box region714 to trigger a return to normal view. Alternatively, if the text entrybox 704 is selected by touching and holding the text entry box region714 on a touch pad on the back of the device, the user may touch thetext entry box region 714 to trigger a return to normal view byreleasing his touch the touchpad to go back to normal operation. Thisprocedure avoids having to bring up the virtual keyboard for text entry.

Although a number of the embodiments described above relate to a devicehaving a front touch screen and a back touch pad, embodiments of thepresent invention may be implemented in devices that utilize only atouch screen.

Hand-Held Device with Two-Finger Touch Triggered Selection andTransformation of Active Elements

According to this embodiment, content rendered on a display of a handheld device may be divided into a number of regions as discussed above.Each region may be associated with a different active element that isrendered on the display. The entire area of the display is divided intoregions that correspond to touch sensitive regions of a touch interface.Each touch sensitive region corresponds to a different active elementshown on the display. Touching the touch interface in a first mode(e.g., with a single finger) operates the touch interface normally.Touching one of the touch sensitive regions in another mode of touch(e.g., with two fingers) activates an enhancement (e.g., transformationor magnification) of the active element corresponding to that region.The first and second modes of operation may be defined arbitrarily.However, in a preferred embodiment, a single finger touch operates thetouch interface normally and a two-finger touch on a screen regioncorresponding to an active element initiates the enhancement of thatelement.

By way of example, and not by way of limitation, content 801, such as aweb page might normally appear on a touch screen 803 as shown in FIG.8A. The content 801 may include active elements, such as a radio button802, a text entry box 804, a link 806 and a check box 808. Inactivenormal text 810 or images may also be displayed. In a conventional modeof operation a single finger touch by operates the screen normally. Forexample, a “swipe” across the surface of the touch screen 803 with afinger F may be interpreted as an instruction to trigger scrolling ofthe displayed content 801.

As discussed above, the area of the displayed content 801 may be brokenup into four different regions as shown in FIG. 8B. These regionsinclude a radio button region 812 that corresponds to the radio button802, a text entry box region 814 corresponding to the text entry box804, a link region 816 corresponding to the link 806, and a check boxregion 818 corresponding to the check box group 808. It is noted thatthere is no region corresponding to the text 810 since the text is notan active element in this example.

A user can then “select” one of the four active elements shown on thetouch screen 803 by touching the corresponding region on the touchscreen with a second touch mode, e.g., a two-fingered touch. Since eachsensitive area is much larger than the displayed active element, theactive elements are easier to select. When a user selects an activeelement by pressing the corresponding region on the touch screen withtwo fingers, the program 210 may interpret this action as an instructionto enhance the corresponding active element, which may then be enhanced,e.g., magnified or transformed, as discussed above. The user can thenmore easily see the active element and interact more easily with itusing the touch screen 803.

For example, as shown in FIG. 8C, if the user presses the touch screenregion 816 for the link with two fingers F₁, F₂ an enhanced link 826 maybe displayed on the touch screen 803. By way of example, the enhancedlink 826 may show a pre-rendered image of the web page or other contentto which the user may navigate by activating the link.

This embodiment facilitates viewing and using the active elements shownon the screen. This also allows for an enhancement of conventional useof a touch screen. Specifically, a two finger touch on one of the activeelement regions of the touch screen 803 may be interpreted as equivalentto touch on a corresponding region of a touch pad on a backside of thedevice. Using two different touch modes, as opposed to two differenttouch interfaces, may simplify the design of a hand-held device andreduce the devices complexity and cost. Using two different touch modes,may also be advantageous even if the device includes both a touch screenand a separate touch pad. The dual mode touch screen may provideadditional flexibility and ease of operation.

In some variations of this embodiment the program 210 may interpret twofingered touch mode actions by tracking the two fingers F₁, F₂independently. For example, if the two fingers F₁, F₂ move in the samedirection, the movement may be interpreted as a “swipe” command. If thetwo fingers F₁, F₂ move in different directions, this movement may beinterpreted as a “pinch” command

There are other variations on this embodiment. For example, two-fingeredtouch may be used to trigger element magnification/transformation andsingle finger touch may be used for scrolling or vice versa. Thisembodiment may also be combined with other embodiments described above,for example, touching a re-purposed region of the touch screen 803outside the portion of the screen that displays an enhanced activeelement may cancel or close the active element. In addition, the conceptin this embodiment may be extended to encompass tracking of three ormore fingers and associating different modes of operation commands withthe number of fingers that are determined to be touching the touchscreen and/or touch pad.

The features of this embodiment may be combined the features of otherembodiments. For example, use of different touch modes may be controlthe degree of magnification of an active element or portion of displayedcontent in the embodiments discussed above with respect to FIGS. 4A-4B.Specifically, the degree of magnification may be correlated to thenumber of fingers used in the mode of touch.

Dynamic Reconfiguration of GUI Display Decomposition Based on PredictiveModel

In this embodiment, content, e.g., a web page rendered on a display, maybe decomposed into a number of regions, each of which is associated witha different active element shown on the display. An entire area of arelated touch interface may be divided into touch sensitive regions,each of which corresponds to a different active element shown on thedisplay. A user may select one of the active elements by touching thecorresponding touch sensitive region. The decomposition may be skewedaccording to a prediction of which active element is likely to beselected next. The prediction may be determined from a predictive modelbased on a history of use of the device 200 by the user. The predictivemodel may be continuously updated as the user uses the device. In someversions of this embodiment, the “skew” of the decomposition may decayover time to a “non-skewed” decomposition that is not based on aprediction. The features of the other embodiments described herein maybe combined with the features of this embodiment.

By way of example and not by way of limitation, as noted above, theprogram 210 may further include a prediction engine 210, which may beconfigured, e.g., by appropriate software programming, to operate thedevice 200 according to a method illustrated generally in FIG. 9A. Asindicated at 902 in FIG. 9A, a portion of content to be displayed on thedisplay 202 may be decomposed, e.g., by Voronoi composition, asdiscussed above. Active elements within the displayed portion may becorrelated corresponding portions of a touch interface, as indicated at904. The touch interface may be the touch pad 204 or the visual display202, if it includes a touch screen. As a user manipulates the touchinterface, the program 210 may optionally determine whether the user hastaken an action as indicated at 906. By way of example, the program 210may detect that the user has selected any portion of the touch interfacethat corresponds to an active element. The program 210 may then adjustproportions of the decomposition of the content shown on the displayaccording to a probability of one or more subsequent actions. Thedecomposition of the content and correlation of the active regions tocorresponding portions of the touch interface may be repeatediteratively over time. By way of example, the prediction engine 221 maycompute probabilities for subsequent actions based on past patterns ofuser behavior following an action of a given type with the device 200.The past behavior may be correlated to a type of content displayed onthe display 102. The probabilities may be updated as the user uses thedevice 200. The screen driver 212 may re-compute the decomposition ofthe displayed portion of the content according to the probabilities asindicated at 908. The size and/or shape of the resulting active regionsof the touch pad 204 may change as a result of the re-computation of thedecomposition.

FIG. 9B illustrates an example of how the decomposition of the displaymay change as probability of subsequent actions change. By way ofexample, as depicted in FIG. 9B, the device may display content such asa web page in response to an initial user action. In this example, thedisplayed content, e.g., a web page 101 displayed on a visual display ofdevice 102 may include active elements, such as a radio button 104, atext entry box 106, a link 108 (e.g., an html link or web link), and ascroll bar 110. The content may also include inactive content, such asnormal text 112 or images. As described above, the area of the displayedcontent page 101 may be decomposed into a radio button region 114corresponding to the radio button 104, a text entry box region 116corresponding to the text entry box 106, a link region 118 correspondingto the link 108, and a scroll bar region 120 corresponding to the scrollbar 110. No region corresponds to the text 112 since, in this example,the text is not an active element. The radio button region 114, textentry box region 116, link region 118, and scroll bar region 120 may bemapped to corresponding regions on a touch sensitive interface.

The prediction engine 221 may determine that, based on passed userbehavior, the user is more likely than not to next use the scroll bar110 than the radio button 104 once the content 101 is displayed.Consequently, the display driver 212 may compute a decomposition of thecontent 101 in which scroll bar region 120 is initially made larger andthe radio button region 114, text entry box region 116 and link region118 may be made smaller than would otherwise be the case if theseregions were determined from a simple unbiased decomposition of content101, e.g. by tessellation of an html canvas. The display driver 212 maycompute the relative areas of the radio button region 114, text entrybox region 116, link region 118, and scroll bar region 120 in accordancewith the relative probabilities that the user is likely to use theseregions within a given time frame.

The likelihood that the user will next use the scroll bar 110 may changeover time. Thus, e.g., as a result of iteration of the decomposition andcorrelation processes, the boundary between the scroll bar region 120and the other three regions may move over time making the scroll barregion smaller and the other three regions larger until the boundary islocated where it would be if determined from a simple unbiaseddecomposition of the content 101. Other variations are possible based onthe amount of information available to the prediction engine about pastuser behavior. For example, as time passes it may become more likelythat the user will use the text box 106. If so, the text box region 116may grow relative to the other three regions. It is noted that thecorresponding probabilities for each active region may decay over timeto an unbiased probability. Consequently, the tessellation of the imagemay decay over time to an unbiased tessellation in such a case.

The features of this embodiment may be mixed with the features of otherembodiments described herein. By way of further non-limiting example,the features of this embodiment may be mixed with the features describedabove with respect to FIGS. 6A-6E. Specifically, the decay of the skewin the decomposition of the displayed content may be applied where anambiguous input on the touch interface is to be disambiguated based onprevious user interaction with displayed active elements as describede.g., with respect to FIG. 6E. In general, the tessellation of thedisplayed content may decay from a biased tessellation in which a touchanywhere on the touch interface is interpreted as an activation of amost recently transformed active element to an unbiased tessellation.Specifically, the decomposition may decay over time from one in whichthe center check box 602C, may be selected or magnified by pressing onone or more of the check box regions 612 that are mapped to the touchscreen 202 to one in which the center check box can only be selected ormagnified by pressing on the portion of the touch interface thatcorresponds to the center check box.

Customization of GUI Layout Based on History of Use

In a variation on the embodiments described above, the layout of contenton a display of a graphical user interface (GUI) may be arranged in apredictive fashion based on a history of use of the GUI. The “layout”may include which items are displayed, where they are displayed, in whatorder they appear, how they appear, and how they work. The layout maydecay to a non-predictive layout over time. The features of the otherembodiments described herein may be combined with the features of thisembodiment.

If the GUI includes a touch interface, the entire area of a touchinterface may be divided into touch sensitive regions, each of whichcorresponds to a different active element. A user can select one of theactive elements by touching the corresponding touch sensitive region.The decomposition may be skewed according to a prediction of whichactive element is likely to be selected next. The prediction may bedetermined from a predictive model based on user behavior.

By way of example and not by way of limitation, as noted above, theprogram 210 may further include a prediction engine 210, which may beconfigured, e.g., by appropriate software programming, to operate thedevice 200 according to a method illustrated generally in FIG. 10A. Asindicated at 1002 in FIG. 10A, a portion of content to be displayed onthe display 202 may optionally be decomposed, e.g., by Voronoicomposition, as discussed above. Active elements within the displayedportion may be correlated corresponding portions of a touch interface,as indicated at 1004. The touch interface may be the touch pad 204 orthe visual display 202, if it includes a touch screen. As a usermanipulates the touch interface, the program 210 may optionallydetermine whether the user has taken an action as indicated at 1006. Byway of example, the program 210 may detect that the user has selectedany portion of the touch interface that corresponds to an activeelement. The program 210 may then adjust the layout of the content shownon the display according to a probability of one or more subsequentactions as indicated at 1008. The adjustment of the content layout andsubsequent decomposition of the content and correlation of the activeregions to corresponding portions of the touch interface may be repeatediteratively over time.

By way of example, the prediction engine 221 may compute probabilitiesfor subsequent actions based on past patterns of user behavior followingan action of a given type with the device 200. The past behavior may becorrelated to a type of content displayed on the display 102. Theprobabilities may be updated as the user uses the device 200. The screendriver 212 may adjust the layout of the displayed portion of the contentaccording to the probabilities. There are a number of different ways inwhich the layout may be adjusted. Preferably, the layout is adjusted ina way that facilitates one or more subsequent actions that are mostprobable. As noted above, this may include adjusting the placement ofactive elements, e.g., by locating active elements that are likely to beused in sequence closer together than in a default layout. In addition,the appearance of the active elements may be adjusted, e.g., activeelements likely to be used may be highlighted or more brightly colored.Furthermore, operation of one or more of the active elements may beadjusted, e.g., the order of items in a list, such as a contact list maybe ordered with the most commonly used items near the top of the list.

By way of example, in a “default” setting, absent any informationregarding past user behavior, a device may display content, e.g., a webpage, containing a radio button 802, text entry box 804, link 806, checkbox 808, and inactive normal text 810 as shown in FIG. 10B. Theprediction engine 221 may determine, based on past user history, thatwhen this page (or similarly configured content) is displayed the userhas a high probability of checking the check box 808 and entering textin the text entry box 804. Based on these probabilities, the displaydriver 212 may modify display of the web page so that the text entry box808 and check box 804 are made larger and/or more prominent and placedin close proximity to each other as shown in FIG. 10B.

In some implementations, the decomposition of the displayed content intoregions and division of the touch interface into corresponding portionsmay change as a result of the modification of the display of thecontent. For example, the display driver 212 may modify the html canvasfor the displayed content. After this happens the touch screen driver212 may perform a new tessellation of the html canvas that the touchscreen driver 213 may use to divide the touch interface intocorresponding portions.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A”, or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

1. A hand-held electronic device, comprising: a case having first andsecond major surfaces; a visual display disposed on the first majorsurface; a touch interface disposed on at least one of the majorsurfaces; a processor operably coupled to the visual display and thetouch interface; and instructions executable by the processor configuredsuch that, when executed, the instructions cause the device to: a)present an image on the visual display containing one or more activeelements; b) correlate one or more active portions of the touchinterface to one or more corresponding active elements in the image onthe visual display; and c) adjust a layout of content shown on thedisplay according to a probability of one or more actions that may betaken with the one or more active elements.
 2. The device of claim 1wherein the instructions are configured to decay the layout to anon-predictive layout over time.
 3. The device of claim 1 furthercomprising a prediction engine configured to compute a probability for asubsequent action based on a past pattern of user behavior following anaction of a given type with the device.
 4. The device of claim 3 whereinthe past behavior is correlated to a type of content displayed on thedisplay.
 5. The device of claim 3 wherein the prediction engine isconfigured to update the probability for the subsequent action as theuser uses the device.
 6. The device of claim 1 wherein the instructionsare configured to adjust the layout in a way that facilitates one ormore subsequent actions that are most probable.
 7. The device of claim 6wherein the instructions are configured to adjust the layout by locatingactive elements that are likely to be used in sequence closer togetherthan in a default layout.
 8. The device of claim 6 wherein theinstructions are configured to adjust an appearance of the activeelements.
 9. The device of claim 8 wherein active elements likely to beused are highlighted or more brightly colored than other elements. 10.The device of claim 6 wherein the instructions are configured to adjustan operation of one or more of the active elements.
 11. The device ofclaim 1 wherein the instructions are further configured to perform atessellation of the image so that the image is divided into one or moreregions that fill the display, wherein each region corresponds to adifferent active element.
 12. The device of claim 11 wherein thetessellation divides the image into one or more convex regions.
 13. Thedevice of claim 12, wherein the tessellation is a Voronoi decomposition.14. The device of claim 11, wherein a size of each region depends on acorresponding probability that a user will use the corresponding activeelement within a given time frame.
 15. The device of claim 14 wherein b)includes making a first region with a first corresponding probabilitylarger than a second region with a second corresponding probability whenthe first corresponding probability is greater than the secondcorresponding probability.
 16. The device of claim 14 wherein theprobability that the user will use the corresponding active elementdepends on a history of use of the device.
 17. The device of claim 14,wherein the probability that the user will use the corresponding activeelement depends on a time elapsed since a previous action.
 18. Thedevice of claim 17, wherein the instructions are configured toiteratively repeat a), b) and c).
 19. The device of claim 18 wherein thecorresponding probabilities for each region decay over time to anunbiased probability whereby the tessellation of the image decays overtime to an unbiased tessellation.
 20. The device of claim 1, wherein thevisual display is a touch screen that includes the touch interface. 21.The device of claim 1 wherein the touch interface is a touch pad. 22.The device of claim 10 wherein the visual display is a touch screen thatis separate from the touch pad.
 23. The device of claim 22 wherein thevisual display is located on first surface of a case and the touch padis located on a second surface of the case that is opposite the firstsurface.
 24. The device of claim 22 wherein the visual display and touchpad are disposed on the same side of the case.
 25. The device of claim22 wherein the visual display and touch pad are disposed on differentsides of the case.
 26. The device of claim 25 wherein the visual displayis disposed on a front side of the case and the touch pad is disposed ona back side of the case.
 27. The device of claim 22 wherein the caseincludes first and second case portions wherein the visual display isdisposed on the first case portion and wherein the touch pad is disposedon the second case portion.
 28. The device of claim 27 wherein the firstand second case portions are slidably connected to each other.
 29. Thedevice of claim 27 wherein the first and second case portions areconnected to each other in a hinged configuration.
 30. The device ofclaim 29 wherein the visual display and touch pad face inward when thefirst and second portions are in a closed position.
 31. The device ofclaim 30 wherein one of the visual display and touch pad faces inwardand the other of the touch pad and visual display faces outward when thefirst and second portions are in a closed position.
 32. The device ofclaim 30 wherein the visual display and touch pad face inward when thefirst and second portions are in a closed position.
 33. The device ofclaim 1 wherein the visual display includes a touch screen and the touchinterface includes the touch screen.
 34. The device of claim 1 whereinthe instructions further comprise instructions that activate one or moreof the elements in response to a distinct mode of touch that is distinctfrom a normal mode of touch that does not activate the one or moreactive elements.
 35. The device of claim 34 wherein the distinct mode oftouch is a two finger touch and wherein the normal mode of touch is asingle finger touch.
 36. The device of claim 1 wherein the instructionsare further configured to activate one of the one or more activeelements in response to a touch to a corresponding one of the activeportions of the touch interface.
 37. The device of claim 1 wherein theinstructions are further configured to present a transformed element onthe display in response to a user interaction with the touch interface,wherein the transformed element interacts with the touch screen in adifferent mode of operation than a mode of operation of thecorresponding active element prior to its transformation.
 38. The deviceof claim 37 wherein the transformed element appears magnified on thevisual display compared to the active element prior to transformationinto the transformed active element.
 39. The device of claim 38 whereinthe instructions are configured to control a degree of magnification ofthe transformed element according to a mode of touch on the touchinterface
 40. The device of claim 37 wherein the instructions furthercomprise an instruction configured to revert the transformed element toa form the active element had prior to being transformed into thetransformed element in response to a signal from the touch interface orafter an interaction with the transformed element is completed.
 41. Thedevice of claim 40 wherein the instructions are configured to revert thetransformed active element in response to a removal of a touch on thetouch interface.
 42. The device of claim 40, wherein the instructionsfurther comprise an instruction to highlight an active element that wasmost recently transformed.
 43. The device of claim 42 wherein the visualdisplay is a touch screen and wherein the instructions are configured tointerpret a touch anywhere on the touch screen as an activation of themost recently transformed active element.
 44. The device of claim 37,wherein the visual display is a touch screen and wherein theinstructions further include an instruction to re-purpose one or moreportions of the touch screen outside the transformed element to act asinputs for commands associated with the transformed element.
 45. Amethod for operating a hand-held electronic device having a case withone or more major surfaces, a visual display disposed on at least one ofthe first major surfaces, a touch interface disposed on at least one ofthe major surfaces, a processor operably coupled to the visual displayand the touch interface; and instructions executable by the processor toimplement the method, the method comprising: a) presenting an image onthe visual display containing one or more active elements; b)correlating one or more active portions of the touch interface to one ormore corresponding active elements in the image on the visual display;and c) adjusting a layout of content shown on the display according to aprobability of one or more actions that may be taken with the one ormore active elements.
 46. The method of claim 45, wherein theprobability of the one or more actions depends on a time elapsed since aprevious action.
 47. The method of claim 45, wherein the probability ofthe one or more actions depends on a history of use of the device. 48.The method of claim 45 wherein the probability of the one or moreactions includes a probability for a subsequent action based on a pastpattern of user behavior following an action of a given type with thedevice.
 49. The method of claim 48 wherein the past behavior iscorrelated to a type of content displayed on the display.
 50. The methodof claim 45 further comprising updating the probability for thesubsequent action as the user uses the device.
 51. The method of claim45 wherein c) includes adjusting the layout in a way that facilitatesone or more subsequent actions that are most probable.
 52. The method ofclaim 51 wherein adjusting the layout includes locating active elementsthat are likely to be used in sequence closer together than in a defaultlayout.
 53. The method of claim 51 wherein adjusting the layout includesadjusting an appearance of the active elements.
 54. The method of claim53 wherein one or more active elements that are likely to be used arehighlighted or more brightly colored than other elements.
 55. The deviceof claim 51 wherein adjusting the layout includes adjusting an operationof one or more of the active elements.
 56. The method of claim 45wherein the instructions are further configured to perform atessellation of the image so that the image is divided into one or moreregions that fill the display, wherein each region corresponds to adifferent active element.
 57. The method of claim 56 wherein thetessellation divides the image into one or more convex regions.
 58. Themethod of claim 57, wherein the tessellation is a Voronoi decomposition.59. The method of claim 58 wherein a size of each region depends on acorresponding probability that a user will use the corresponding activeelement within a given time frame.
 60. The method of claim 59 whereinthe corresponding probabilities for each region decay over time to anunbiased probability whereby the tessellation of the image decays overtime to an unbiased tessellation.
 61. The method of claim 45, furthercomprising activating one of the one or more active elements in responseto a touch to a corresponding one of the active portions of the touchinterface.
 62. The method of claim 45, further comprising transformingone or more of the active elements to a transformed element in responseto a user interaction with a corresponding one or more of the activeportions of the touch interface.
 63. The method of claim 62, wherein thetransformed element interacts with the touch interface in a differentmode of operation than a mode of operation of the corresponding activeelement prior to its transformation.
 64. The method of claim 62 whereinthe transformed element appears magnified on the visual display comparedto the active element prior to transformation into the transformedactive element.
 65. The method of claim 64 wherein a mode of touch onthe touch interface controls a degree of magnification of thetransformed element.
 66. The method of claim 62, further comprisingreverting the transformed element to a form the active element had priorto being transformed into the transformed element in response to asignal from the touch interface.
 67. The method of claim 66 wherein aremoval of a touch on the touch interface triggers reverting thetransformed element.
 68. The method of claim 62, further comprisinghighlighting an active element that was most recently transformed. 69.The method of claim 68 wherein a touch anywhere on the touch interfaceactivates the most recently transformed active element.
 70. The methodof claim 45, further comprising, for each active element of the one ormore active elements, determining the corresponding probability that auser will use the active element within a given time frame.
 71. Acomputer readable medium programmed with computer executableinstructions for operating a hand-held electronic device having a casewith one or more major surfaces, a visual display disposed on at leastone of the first major surfaces, a touch interface disposed on at leastone of the major surfaces, a processor operably coupled to the visualdisplay and the touch interface, wherein the instructions are executableby the processor to implement a method comprising: a) presenting animage on the visual display containing one or more active elements; b)correlating one or more active portions of the touch interface to one ormore corresponding active elements in the image on the visual display;and c) adjusting a layout of content shown on the display according to aprobability of one or more actions that may be taken with the one ormore active elements.